\label{ch:evaluation}

This chapter presents the experimental results of the simulation. All of the experiments were performed using the simulator briefly discussed in Section~\ref{sec:simulation}. The organization of this chapter is as follows: In Section ~\ref{sec:evalapi} unique features provided by the API are summarized. Section ~\ref{sec:expdesign} discusses about the design of the experiment. In Section ~\ref{sec:controlqos}, various QoS parameters such as response time and average outgoing bandwidth are evaluated in respect of CyberOrgs-MMOG implementation. Mechanisms provided by CyberOrgs-MMOG API enables multiple peers to coordinate resources. Section ~\ref{sec:performance} presents the performance analysis of the API mechanisms.

\section{CyberOrgs-MMOG API}
\label{sec:evalapi}
CyberOrgs-MMOG API comes with a library that allows integrating resource control mechanisms for Massively Multiplayer Online Games. Following are some of the features come with the API.

\begin{itemize}
\item {\emph{Improved programmability:}}
The API allows game developers to integrate sophisticated functionalities, which adopts to existing proven models and approaches. By properly configuring the API developers can build scalable MMOGs without compromising the game performance. However, when integrating the API-supports to an existing MMOG, developers have to make sure that the game design allows spatial distribution of the game world and provides enough abstraction so that the API can encapsulate and control the distribution of game computation.

\item {\emph{Resource coordination mechanisms:}}
The API supports mechanisms that allows acquisition of resources during a overload condition in a flexible and seamless way. Similarly, mechanisms to release resources are also integrated within the API. The addition and release of resources can dynamically control the game performance even in case of hour-to-hour fluctuations. The results on the evaluation of these API mechanisms are discussed later in this chapter.

\end{itemize}

\section{Experiment Design}
\label{sec:expdesign}
In order to demonstrate the mechanisms supported by the CyberOrgs-MMOG API, the experiment is designed to show that the quality of service requirements can be fulfilled without significant overhead. Two of these performance parameters are evaluated in this experiment - average response time and average outgoing bandwidth. 

Response time represents the delay that a player experiences when sending a request to a server. In short, response time is the time difference between a player's action and a server's response to that action. It is expected that the average response time meets the requirement of the game's rendering rate. The game client should be able to render in such a rate so that a player does not experience any lag. The mechanisms supported by the CyberOrgs-MMOG API facilitates acquisition of resources whenever the response time goes beyond a certain limit and vice versa. Similarly, average outgoing server bandwidth can be controlled. Usually, for a multiplayer game server the outgoing data packets contain more data than incoming requests from a game player. Therefore, outgoing bandwidth plays a vital role in keeping a high-performance gameplay. 

In the following section, these two performance parameters are evaluated to show that these parameters can be used to control the game performance. The overhead caused by these mechanisms are analyzed in Section ~\ref{sec:performance}. The overhead is observed by comparing the average response time and outgoing server bandwidth with those of a server not integrated with the CyberOrgs-MMOG API. Each of these experiments was performed for up to 20 runs and averaged.

\section{Controlling QoS Parameters}
\label{sec:controlqos}
This section presents the evaluation of game performance parameters by coordinating resources dynamically. The experiment is done with two parameters- average response time and average outgoing bandwidth among servers. At first, each of these parameters are evaluated based on the number of players.  A new resource is added or released to the game when the number of players reach a certain threshold. Then we evaluate by defining a cut-off value for these parameters. Following subsections present the results.

\subsection{Average Response Time}
\label{subsec:response}
Figure ~\ref{fig:response} and ~\ref{fig:rresponse} show the change in response time against the number of players. A new resource is added when the average number of players per server reaches 50, in other words, when the total number of players reaches a multiple of 50. In these figures, black marks represent addition or release of resource. In this experiment, new players are distributed among the resources in an uniform fashion. From the figure, we can see the response time rises up to 70ms and each time a new resource is added an improvement in response time is observed. Moreover, we can observe from the figure that the peak response times are not equal, though the average load among all servers is the same. The reason behind this can be a resource is loaded by other applications. Another reason can be the uneven distribution of players among the servers as the simulated players can move from one zone to another during a migration (resource addition). On the other hand, Figure ~\ref{fig:rresponse} shows the release of resources in response to decrease in number of players. The experiment starts with 200 simulated players uniformly distributed among 4 resources. Similar to the previous experiment, resources are released when the number of players decreased to a multiple of 50. In this case, the minimum average response time observed is approximately 20ms.

\begin{figure}
\centering
\includegraphics[width=150mm, height = 100mm]{response}
\caption{Resource added in response to increase in number of players.}
\label{fig:response}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=150mm, height = 100mm]{rresponse}
\caption{Resource released in response to decrease in number of players.}
\label{fig:rresponse}
\end{figure}

Figure ~\ref{fig:response} and ~\ref{fig:rresponse} show addition or release of resources based on the number of players. However, this approach does not keep the increase or decrease in response time under control. By controlling the response time a game developer can control the quality of the gameplay. Figure ~\ref{fig:response_cutoff} and ~\ref{fig:rresponse_cutoff} show the results of such control in response time. In this experiment, a new resource is added or released only when the response time reaches certain values (can be defined in the configuration file described in ~\ref{sec:conf}), which in this case are 60ms and 20ms, respectively. These threshold values are set based on the minimum time required to render a frame. According to some studies,\footnote{http://www.grand-illusions.com/articles/persistence\_of\_vision/} the required minimum frames per second is highly dependent on the type of animation in concern. However, often 24fps (frames per second) is used as a standard value in many computer games. This gives the Graphics Processing Unit (GPU) around 42ms time to render and display a frame. Based on this observations, the upper limit for response time is set to 60ms to detect an overload condition. On the other hand, 20ms of response time is set as a minimum requirement while releasing resources. We allowed this space in response time to avoid fluctuations which often develops due to network delays. It is also observed that the number of players may not be the same for every addition or release of resource. Another observation is that the peaks do not happen when a resource is added rather  after a few more players are added. This is because the simulator application continuously adds new players in a random time interval of 0s to 20s, which often may occur before a migration (resource addition) process is complete.

\begin{figure}
\centering
\includegraphics[width=150mm, height = 100mm]{response_cutoff}
\caption{Resource added in response to increase in response time.}
\label{fig:response_cutoff}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=150mm, height = 100mm]{rresponse_cutoff}
\caption{Resource released in response to decrease in response time.}
\label{fig:rresponse_cutoff}
\end{figure}

\subsection{Average Outgoing Bandwidth}
\label{subsec:bandwidth}

 \begin{figure}
\centering
\includegraphics[width=150mm, height = 100mm]{bandwidth}
\caption{Resource added in response to increase in number of players.}
\label{fig:bandwidth}
\end{figure}

Figure ~\ref{fig:bandwidth} and ~\ref{fig:rbandwidth} show the change in average outgoing server bandwidth per server against the number of players. As in Subsection ~\ref{subsec:response}, similar setup is used in this experiment. A resource is added or released when the total number of players reaches multiple of 50 and the change in average server bandwidth is observed. As we can see from the Figure ~\ref{fig:bandwidth}, the average bandwidth rises up to 110KB/s and each time a new resource is added an immediate improvement in average server bandwidth is observed. Figure ~\ref{fig:rbandwidth} shows that the bandwidth goes down to 20KB/s while releasing the resources, We also observe that the peak bandwidths are not equal, though the average load among all servers is the same. Similar reasoning as discussed in Subsection ~\ref{subsec:response} can also be applied to explain this observation.

 \begin{figure}
\centering
\includegraphics[width=150mm, height = 100mm]{rbandwidth}
\caption{Resource released in response to decrease in number of players.}
\label{fig:rbandwidth}
\end{figure}
 
To keep the bandwidth under control, cutoff values are set to 80KB/s and 30KB/s and the results are shown in Figure ~\ref{fig:bandwidth_cutoff} and ~\ref{fig:rbandwidth_cutoff}, respectively. As the available outgoing bandwidth is a network resource which can vary from one server to another, these values are set based on the previous approximation of bandwidth requirement when number of players were considered as a control parameter (Figure ~\ref{fig:bandwidth} and ~\ref{fig:rbandwidth}). This approach allows the resource coordination decisions to be made only when the average bandwidth exceeds this threshold. By controlling the bandwidth, a game owner can control the network delay experienced by a game player. Same as the experiment with the response time, the threshold bandwidth can be set in the configuration file discussed in Section ~\ref{sec:conf}. 
 
\begin{figure}
\centering
\includegraphics[width=150mm, height = 100mm]{bandwidth_cutoff}
\caption{Resource added in response to increase in outgoing bandwidth.}
\label{fig:bandwidth_cutoff}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=150mm, height = 100mm]{rbandwidth_cutoff}
\caption{Resource released in response to decrease in outgoing bandwidth.}
\label{fig:rbandwidth_cutoff}
\end{figure}

It is possible to enforce cutoff values for both response time and bandwidth at the same time. A game owner can choose to control one of these parameters or both. Moreover, other server controlled quality of service parameters can be defined with some modification in the CyberOrgs-MMOG API.

\section{Performance Analysis}
\label{sec:performance}

Experiments have been carried out to look at the overhead of using the CyberOrgs-MMOG API. Instead of adding new resources, we observe the change in average response time with increase of players. The data observed are compared with a similar setup without using the CyberOrgs-MMOG API. This section also presents experimental results on the time the API takes to coordinate resources.

\subsection{Overhead Analysis}
\label{subsec:overhead}
Figure ~\ref{fig:response_compare} shows the comparison of response times with or without the use of CyberOrgs-MMOG resource coordination mechanisms. Two separate experiments are carried out. Each of them are executed for 20 runs and averaged for more accurate results. For the experiment using CyberOrgs-MMOG API, all resource coordination and acquisition mechanisms are enabled except that players were not allowed to migrate. And then the result is compared with the response time provided by the game itself. It is observed that the response times provided by both experiments are very similar. The game with CyberOrgs-MMOG API is configured to make resource decisions when the number of players is more than 30. This triggers the load manager and the zone manager to look for additional resources. 

\begin{figure}
\centering
\includegraphics[width=150mm, height = 100mm]{responseCompare}
\caption{Effects on average response time of using CyberOrgs-MMOG API.}
\label{fig:response_compare}
\end{figure}

Figure ~\ref{fig:bandwidth_compare} presents the result of similar experiment on the outgoing bandwidth. It is observed that the bandwidth requirement is little higher than the game without using the API, specially when the number of players is more than 30. Same as the previous experiment, the game with CyberOrgs-MMOG is configured to make resource decisions and coordinations when the number of players reaches 30. The higher bandwidth requirement is the result of coordination with other resources, migration of computations and synchronization between the zone boundaries.  

\begin{figure}
\centering
\includegraphics[width=150mm, height = 100mm]{bandwidthCompare}
\caption{Effects on outgoing bandwidth of using CyberOrgs-MMOG API.}
\label{fig:bandwidth_compare}
\end{figure}


\subsection{Analysis of resource coordination mechanisms}
\label{sec:delayanalysis}
In this section, time taken to execute resource coordination mechanisms is analyzed. These mechanisms include detection of an overload/under load condition, searching for a suitable resource and migration of the computation (a part of the zone). The experiment is done by configuring the game to enable migration for varying zone sizes. Each of these migrations is performed 20 times and averaged.  

Figure ~\ref{fig:mig_time} shows the results of this experiment. Dynamic resource acquisition process consists of three mechanisms provided by CyberOrgs-MMOG API. Mean time and standard deviation for each of these mechanisms - Detection of a load condition, Search for new resource and Migration to another resource are shown. The number of players in the zone that is being migrated is also shown.

\begin{figure}
\centering
\includegraphics[width=160mm, height = 130mm]{migrationtime}
\caption{Time taken to execute resource coordination mechanisms.}
\label{fig:mig_time}
\end{figure}

Table ~\ref{tab:migrationtime}  presents the time taken for the resource coordination mechanisms with respective standard deviations. It is observed that the mean time taken for detecting a load condition (over/under load) does not vary significantly with the size of the zone. Therefore, it may be safe to assume that the time taken for a detection is not dependent on the number of players in the zone. We observe similar characteristics for search operation. However, in this experiment, the directory manager is located in the same network as the broker and game server.  In a real world application, where there can be thousands of available resources to search from and with much greater geographical distances between brokers, resources and directory managers, these values can rise quite significantly. On the other hand, time taken to migrate a number of players along with other entities from one server to another increases non-linearly with the increase of players. When a zone migrates, the CyberOrgs-MMOG API allows migration of entities one by one. Therefore, it is quite evident that the more the number of players and entities the more time it takes. This has been done intentionally to reduce the time lag that occurs during migration. Besides, the bigger the size of the game zone the more data needs to be synchronized between the servers. While the other two operations do not show any correlation with the number of players/entities, for migration, the time increases non-linearly with the increase of players. One observation from the Table ~\ref{tab:migrationtime} is zones with sizes 125 and 150 have a smaller difference in migration times than other zone size differences. There could be several reasons behind this. One reason is that when moving zones from one host to another the number of surrounding entities could differ significantly as entities such as the enemies can move in or out of the zone during a migration. However, the actual migration time does not have any effect on the response time and thus, does not affect the game performance. Though this may affect the bandwidth requirement as during these migration periods servers have to be kept synchronized. The analysis provided in this subsection can be used to decide on the minimum size of the zones based on the availability of bandwidth.

\begin{table}
  \centering
  \begin{tabular}{ | c | c | c | c | c | c | c | c |}
    \hline
    \# of players & \multicolumn{2}{c|}{Detection} &  \multicolumn{2}{c|}{Search} &  \multicolumn{2}{c|}{Migration}  & Total\\ \cline{2-7} 
    & mean & stddev & mean & stddev & mean & stddev &            \\ \hline
50 & 110 & 11.2 & 320 & 23.2 & 981 & 103.4 & 1411 \\ \hline
75 & 105 & 13.7 & 344 & 14.8 & 2911 & 167.7 & 3360 \\ \hline
100 & 123 & 6.6 & 325 & 22.5 & 5438 & 234.5 & 5886 \\ \hline
125 & 118 & 9.2 & 336 & 19.6 & 9055 & 255.6 & 9509 \\ \hline
150 & 125 & 12.7 & 328 & 15.5 & 9633 & 271.9 & 10086 \\ \hline
175 & 112 & 9.7 & 334 & 16.7 & 15264 & 563.4 & 15710 \\ \hline
200 & 125 & 5.5 & 338 & 20.2 & 28586 & 989.3 & 29049 \\ \hline
  \end{tabular}
  \caption{Time taken to execute resource coordination mechanisms.}
\label{tab:migrationtime}
\end{table}


\section{Chapter Summary}
\label{sec:summary}
In this chapter, experimental results in terms of quality of services are presented. The effects on the game performance by the mechanisms provided by CyberOrgs-MMOG API are also evaluated. We notice  improvement in response time per player as well as average outgoing bandwidth per server when resource is dynamically added or released based on a cutoff value. This approach provides much more flexibility and control over the game performance. It is also observed that supporting these mechanisms do not cause a significant overhead compared to the performance of the gameplay without the API. We also notice that the number of players do not affect load detection and resource search operations though it has a non-linear correlation with the migration operation.



